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Abstract 



We investigate the performance of sparsely-connected networks of integrate- 
and-fire neurons for ultra-short term information processing. Wc exploit the 
fact that the population activity of networks with balanced excitation and inhi- 
bition can switch from an oscillatory firing regime to a state of asynchronous 
irregular firing or quiescence depending on the rate of external background 
spikes. 

We find that in terms of information buffering the network performs best 
for a moderate, non-zero, amount of noise. Analogous to the phenomenon of 
stochastic resonance the performance decreases for higher and lower noise lev- 
els. The optimal amount of noise corresponds to the transition zone between a 
quiescent state and a regime of stochastic dynamics. This provides a potential 
explanation on the role of non-oscillatory population activity in a simplified 
model of cortical micro-circuits. 
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1 Introduction 



The brain processes information in a constantly varying world. The visual 
scenes one sees in everydaylife are extremely rich and change rapidly in time. 
In addition, the human eye performs more than three saccades every second 
causing sudden changes in the visual input [53]. Complex sounds, such as 
speech or music vary continuously in time and frequency. Moreover, the human 
brain has to deal with simultaneous sound sources from different origins, that 
are superimposed and mixed together. From such a perspective, it is clear that 
cortical areas are confronted with time-varying sensory inputs rather than 
stationary stimuli. While attractor neural networks are considered suitable 
as models of working memory [67,22,1], they are less useful to explain ultra- 
short memory buffer properties in the range of 10 to 100ms of signals that 
vary continuously in space or time. Recently, models of continuous information 
processing in recurrent neural networks have been proposed under the names 
of Liquid State Machine (LSM, [45]) and echo state networks [31,32], both 
related to the timing network of Buonomano and Merzenich [9]. Those models, 
that perform computation using transient activity (as opposed to convergence 
to a stable state), are sometimes referred to as models of computation with 
dynamic states. The idea underlying those models is that the instantaneous 
state of the network provides a rich reservoir of non-linear spatio-temporal 
transformations of the inputs. 

In the framework of the liquid state machines, learning only acts on the read- 
out structures, the network itself remaining fixed. Several recent studies focus 
either on the biological realism of such model networks or on possible imple- 
mentations of this approach in machine learning or robotics. The role played 
by dynamical synapses was studied from the perspective of efficient tempo- 
ral processing in [50,51]. Principles of liquid state machines were applied to 
the analysis of a variety of time series [49,52] for computer vision [44], move- 
ment generation and control [33,39], and prediction of chaotic time series [32]. 
The idea of performing (simple) computations based on perturbation of a real 
liquid (water) was investigated in [21]. 

Cortical micro-circuits are extremely complex recurrent networks of neurons. 
A given neuron is functionally connected to only a relatively small fraction 
of the other neurons. The connection strength between neurons is not fixed, 
but is mediated by synapses that have their own dynamics [3,56,15,19,28,60]. 
Maass and colleagues have chosen to simulate networks with a detailed set of 
biologically-inspired parameters. They implemented distance-dependent con- 
nectivity and different refractory periods and thresholds for the inhibitory 
and excitatory pools. They also investigated the role of dynamic synapses and 
introduced stochasticity in the values of some parameters [45]. 
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On the other end of the spectrum of ncTiral network models are ideahsed net- 
works that replace real cortical connectivity by sparse random connections and 
reduce neuronal diversity to two neuron types, i.e. excitatory and inhibitory. 
Even from these simplified models, a rich dynamics can emerge such as fast 
and slow oscillations, synchrony and even chaotic behaviour [5,7,29,62,66]. 
These simplified models have the advantage that theoretical methods for in- 
vestigating the dynamics are available via an analytical approach, so that 
the parameter space can be explored systematically. Based on methematical 
analysis, the network behaviour can be classified in a small number of types, 
such as fast oscillations, slow oscillations, or spontaneous asynchronous activ- 
ity [5,20], a rigorous classification which would be difficult to obtain by pure 
simulation-based studies. 

A remarkable result of such studies concerns asynchronous activity [5,7]: in 
the phase diagram of the activity of sparsely-connected network of inhibitory 
and excitatory neurons, there exists a stable phase of asynchronous irregular 
firing in which the overall activity is stationary (its statistical properties such 
as mean and variance are time-independent) but the activity of individual neu- 
rons is highly irregular. From the point of view of information processing in a 
cortical-like network, such a regime of firing provides very interesting proper- 
tics. In particular, we can imagine that information about past input can be 
buffered in the perturbations of the stable state of asynchronous activity. 

The presence of such a memory trace of past stimulations is hypothesised for 
example in models of conditioning (such as reinforcement learning [64]) under 
the name of eligibility trace. However the underlying mechanism for such a 
trace is not known, even though many models have blossomed in the past 
decade. In most models (for example [63]), the clock is replaced by a series 
of spectra without any strong biological relevance. Bullock and al. [4] have a 
model of a clock that is dependent on the Ca^+ activity and Contreras-Vidal 
[13] hypothesised an activation of different subsets of striosomes. However 
these mechanisms are supposed to cover many orders of magnitude in timing 
(from a fraction of a second up to a few seconds), and it is unlikely that such 
a broad spectrum can be covered by a single ionic mechanism. Another way 
of keeping track of time is to have a recurrent neural network, that will store 
dynamically the timing information. Such a mechanism is likely to exist in the 
olivo-cerebellar system [34]. 

In this paper we make use of the mathematical analysis of sparsely-connected 
networks [5] in order to relate macroscopic states of the network with evalua- 
tions of information buffering, thus establishing a link between network theory 
and the liquid state machines [45]. Specifically, we inject a time-dependent in- 
put current into a network of excitatory and inhibitory neurons with sparse 
connectivity. We want to know for how long information about an input signal 
is kept in the network. In order to get a quantitative answer, we attempt to 
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reconstruct the input signal at time t — T using the instantaneous membrane 
potentials of all neurons at time t. If the error of signal reconstruction is small, 
we say that the network is capable of "buffering" information for a time T. 



2 Model and Methods 



2.1 Integrate- and- fire neuron 

The system we study is a sparsely connected network of leaky integrate- and- 
fire (IF) neurons. Our network is composed of n = 200 IF neurons, 80% 
of which are excitatory and 20% inhibitory. The equation for the membrane 
potential of such neurons can be written as: 

TmUiit) = -Ui{t) + RIi{t) (1) 

where Jj is the total input, = RCm the effective membrane time constant, R 
the effective input resistance and Cm the membrane capacity. The total input 
of neuron i can be decomposed into contributions of the presynaptic spikes 
from other neurons j within the micro-circuit under consideration, an external 
drive J^"^* and a noise term /""■^^ that models background spikes (described in 
detail in section 2.2) from other brain areas that are not described explicitly: 

Rm = Tmj2 E - + Rir\t) + Ri~{t) (2) 

where Mj is the ensemble of presynaptic neurons, the time neuron j fires 
its /c'th spike and D is a transmission delay. Synaptic weights are chosen from 
two values, either excitatory, Uj = u>e or inhibitory, cuj — —u>i. Thus from 
equations (1) and (2), the EPSP's resulting from spike arrivals at an excitatory 
synapse are of amplitude ue and decay with the membrane time constant r^. 
Analogously, the IPSP's resulting from spike arrivals at an inhibitory synapse 
are of amplitude —cui and decay with the membrane time constant r^. The 
total depolarisation of neuron i is the sum of all EPSP's and IPSP's that have 
arrived since the last spike of postsynaptic neuron i. 

When the depolarisation Ui{t) of neuron i reaches a threshold 6, a spike is 
emitted and the membrane potential is reset to a potential Ureset after an 
absolute refractory period r^p during which the neuron is insensitive to any 
stimulation. The numerical values are shown in table 1. 
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Neuron parameters 



Network parameters 



Membrane time 
constant 


Tm 


20 ms 


Threshold 


e 


5 mV 


Absolute re- 
fractory period 




2 ms 


Transmission 
delay 


D 


1 ms 


Reset mem- 
brane potential 


'Preset 


mV 


Effective input 
resistance 


R 


10 


Membrane ca- 
pacity 


Cm 


2 nF 



Number of neu- 
rons 


n 


200 


Connectivity ratio 


e 


0.2 


Excitatory synap- 
tic efficacy 




1 mV 


Inhibitory synap- 
tic efficacy 




5 mV 


Number of ex- 
ternal excitatory 
neurons 


Ne 


100 


Spiking freq. of 
the external ex- 
citatory neurons 
(model A) 


^exc 


10-^-100 Hz 



Tabic 1 

Left. Numerical values of the model neurons (leaky Integrate-and-Fire neuron) used 
for the simulations. Voltage is measured with respect to the resting potential. Right. 
Numerical parameters of the simulated neural network. The connection probability 
of one neuron to another corresponds to the connectivity ratio e. 

2.2 Stochastic background input 



We consider two noise models. The first model assumes stochastic arrival of 
excitatory spikes from other areas. Since a complete analytical description of 
the impact of this noise model on the dynamics of the network has been done 
[5], we can connect our results to the known macroscopic dynamics of the 
randomly connected networks. 

In a second noise model, we include both excitatory and inhibitory spikes 
from outer regions. The combination of inhibition with excitation generalises 
the first model and permits change in the fiuctuations of the drive without 
changing its mean. It also allows us to draw an interesting link to a physical 
phenomena known under the name of stochastic resonance. 



2.2.1 Model A: Stochastic spike arrival at excitatory synapses 

In order to simulate the intense synaptic bombardment of neurons observed in 
vivo (for a review see [14]), we first consider purely excitatory stochastic spike 
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arrivals coming from A^^; external excitatory neurons. We assume that each 
spike changes instantaneously the membrane potential of the post-synaptic 
neurons (satisfying equation (1)). Stochastic spike arrivals can be described 
by the sum of Ne homogeneous Poisson processes of rate Uexc- In such a 
description the probability of having a spike in a time window is stationary 
in time. There is no correlation between spikes, i.e. the occurrence of a spike 
at a given time does not influence the future. If we make the approximation 
that a neuron receives a large number of presynaptic contributions per unit 
time, each generating a change in the membrane potential that is relatively 
small compared to the firing threshold 9, the noise term can be written as a 
constant drive plus a fluctuating part ^ : 

Rir'%t) = iiE + 'JEV^Viit) (3) 

where i^e = NEUiEi^excTm and cr^ = oje\/ NEi'excTm- By ?7, we denote standard 
Gaussian white noise of zero mean and unit variance, i.e., 

< r]i{t) >= 
< ViitMt') >= S{t - t') 

This type of excitation correspond to the analysis of asynchronous activity in 
sparsely connected network of IF neurons made in [5]. The results from our 
simulations can thus be directly interpreted with the phase diagram of the 
activity, figure 2 in reference [5]. 

We will focus on the influence of the external spiking activity to the perfor- 
mance of the network in the range 10^^ Hz < i^eic < lOOHz. 

2.2.2 Model B: Excitatory and inhibitory stochastic spike arrivals 

In the second noise model we assume stochastic spike arrival from both excita- 
tory and inhibitory neurons. Let us consider that we have an external popula- 
tion Ne of excitatory neurons firing at a rate Ugxc, as in model A. We now add 
a further input population (referred thereafter as the "balanced population") 
made out of two groups of neurons, one of them excitatory and the other one 
inhibitory. In order to have a balance between excitation and inhibition, we 
choose to have five times as many excitatory neurons as inhibitory neurons, 
but with inhibitory synapses that are five times as strong, i.e., Ne^ = ^Nf^, 
with ^ = 5. This approximate balance between excitation and inhibition is 
thought to take place at a functional level in cortical areas [41,59]. The choice 
we made to have an exact balance in this additional population of neurons 
is for the sake of simplicity only. Its mean contribution is therefore zero and 
a change in the firing rate of this population only affects the variance of the 

^ This substitution is known as a diffusion approximation, see e.g. [5] 
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drive. In addition, modulation of the variance part of the drive wiU allow us 
to focus on pure noise effects for a fixed mean drive. 

With such a model we can write the noise term, similar to the previous model 
A: 

Rir^'it) = //B + apopV^Viit) (4) 
where we have defined /ie — NEUJE^exc^m and 

The effective mean drive corresponds to the mean excitation from the purely 
excitatory population only since the mean contribution of the balanced popu- 
lation is zero . The variance can now be varied independently of the effective 
mean input, by changing the discharge rate z/pop of the balanced population. 

The noise terms in both models (A) and (B) can be expressed as: 

Rir''{t) = fiE + a^rji{t) 

where cr = (Je in noise model (A) and a — apop in noise model (B). Since 

the drive ^e = NEOOE^excTm > corresponds to the background activity of 
an excitatory population, the mean drive is always positive. For A^^; = 100, 
ue = ImV, Pexc = ^Hz and = 20ms we get a mean depolarisation of 
fiE = 2mF due to the noise input. 

2.3 Network structure 

Both excitatory and inhibitory neurons are modelled with a membrane time 
constant of 20ms. The neurons are weakly (connection probability = 0.2) 
and randomly connected through simple static synapses. Wc carefully chose 
the synaptic strengths, tui = Bloe so that for i/^xc = 2 — 5Hz and Ne = 100, 
the network is in a regime of asynchronous and irregular firing (see figure 3B 
centre), based on the phase diagram described in [5]. In such a regime, there 
is a stationary overall activity and highly irregular spike firing of individual 
cells. Specifically, we take r^p = 2ms, D = 1ms, 6 = 5mV, Uj-eset = Om^/, 
uje = ImV and uj = 5mV (see table 1) , but other combinations of parame- 
ters give qualitatively similar results. An extensive scanning of the parameter 
space has been carried out ^ . This robustness originates from the fact that the 
phase diagram of the type of networks we study in this article, can be quali- 
tatively drawn as a function of two parameters only; the external drive (mean 
and variance) and the effective ratio between excitation and inhibition. The 

^ For example, the precise values of the synaptic strengths affect the exact location 
of the optimal performance when larger than O.lmV but not the overall trend of 
the results. 
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presence of an asynchronous phase of neural activity (and therefore the essence 
of our results) is guaranteed for a broad range of the neuron's parameters [5]. 

With a stronger input, the system reaches a phase of synchronous firing with 
fast oscillations [5]. For weak drive, activity will tend to zero (quiescent state) 
since the network itself is not capable of sustaining intrinsic activity. 

As we will see later, the absence of synchrony at the working point of the 
network is important in order to let information about a past stimulation flow 
for a long time in the network. 

2.4 Evaluating information buffer properties 

We perform simulations of the network with some time- dependent inputs 
Ii^^{t). The question we want to ask is: how much information is left about 
the external inputs after some time T ? Given some continuous input If^^{s) 
we expect that the instantaneous state of the network (given by its membrane 
potentials {uj{t)\l < j < n}) holds information about If^^{s) for s < t. If 
a reconstruction of the input signal /f^*(t — T) can be achieved by looking 
at the membrane potentials at time t, we say that the network can "buffer" 
information for a time T. 

We assess the information buffering capacity of the network with a procedure 
analogous to [45] and [31]. We inject simultaneously A^^ independent input 
signals to A^^ disjoint groups of randomly chosen neurons (see figure lA), every 
neuron receiving exactly one input; e.g. for Ng — A input signals we have 4 
groups of 50 neurons in our network of 200 neurons. Ng readout structures 
have access to the membrane potentials of all neurons in the network at time 
t and use this information to estimate the input signal that was present at 
time t — T (see figure IB). 

We consider two different methods: 

• Linear read-out. 

The outputs of the readout structures are simple linear combinations of 
all the membrane potentials i.e. 

Outputit) = "^akUkit) + ao = ■ il+ ao 

k 

where u — {ui,U2, ...,Un)'^ is the vector of membrane potentials and ckq a 
bias term. 

• Non-linear kernel-based read-out. 

In order to know whether more complex read-out structures can extract 
significantly more information than the linear read-out, we used Support 
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Vector Machines (SVM's) [10] , a powerful method from the field of sta- 
tistical learning [65]. It belongs to the family of kernel-based methods [58]. 
In those techniques, the input is sent to a high-dimensional space in which 
linear fits are expected to work, i.e. 

Output{t) — ^ ai)C{u{t) , u{ti)) + ao 
I 

The sum runs over several samples of instantaneous network states u{ti). 
The set of relevant samples is selected by the SVM algorithm. Different 
kernels /C(-) can be used in order to implicitly build a multi-dimensional 
space. In section 6 we will compare the linear read-out to a Support Vector 
Machine with a Gaussian kernel. We used SVM-torch [12] for the SVM 
method. 

Only the parameters (the weights ak) of the readout structures are tunable, 
while the network of IF neurons itself remains fixed. Since we want to assess 
how much we can learn about the input at time t — T if we read out the 
membrane potentials at time t, we minimise the error : 



E{d) = — J2[0utput{t) - Input{t - T)f 
2A J 

with A = Y^t{Input{t) — J)^ where I denotes the mean drive. Minimisation 
is achieved by an optimisation procedure (optimal regression for the linear 
read-out) on the training set. Later in the results section, we will refer to this 
error as the signal reconstruction error. The Input{t — T) plays the role of a 
target value for optimisation. For the linear readout there is a unique optimal 
set of weights {at} for every delay . After a training period, the weights of 
the readout structures are frozen and Ng new input signals are introduced in 
the network. Outputs of the readout structures are then compared with their 
corresponding targets. For all the simulations, the trajectories used for training 
correspond to 50'000 time steps of simulation. Performance is measured on an 
independent test set of 50'000 time steps. For the sake of clarity, all figures 
in the result section are obtained for a single input iV^ = 1. In the discussion 
section, we vary the number Ng of inputs. 



^ The application of the projection theorem [43] say that it is true if the number n 
of free parameters (the number of neurons, n = 200 in our case) is smaller than the 
number of independent examples. We took 50'000 examples, but these are slightly 
correlated in time. A realistic measure is to divide the number of examples by the 
effective autocorrelation of the input (3.3ms up to 80ms, see section 2.5). Even in 
the worse case, a lower bound for the number of linearly independent examples is 
^°gQ^° = 625 > n = 200. The lengths of the simulations guarantees that there is a 
unique optimal set of weights. 
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A 



B 




SI 



U(S1)20 



U(Sx) 



sx fn/vyv^ 




Fig. 1. A: The different signals Sl,..,Sx are introduced to randomly chosen inter- 
connected neurons. Every neuron receives exactly one input from the exterior. Only 
two input signals SI and Sx are shown in the figure. B: Based on the momentary 
state of all membrane potentials (those receiving directly the signal SI and those 
receiving any other signal Sx) , a readout structure is trained to guess the amplitude 
of its corresponding input a time T before (referred as delay thereafter). 



2.5 Autocorrelated inputs 



As mentioned before, all neurons in the network receive one of A^^ input signals 
If^^{t). We build an input such that its autocorrelation profile depends on a 
single parameter and decays to zero beyond a given value. In order to do so, 
the total simulation time is broken into segments of duration T^ax- During 
each segment of length T^ax the input is kept constant. At the transition to 
the next segment, a new value of /f^*(t) is drawn from a uniform distribution 
centred at zero, between -20 pA and +20 pA (-7.5 pA and -1-7.5 pA as an 
additional example shown in figure 2B). This procedure results in a triangular 
autocorrelation profile A{s) that is zero for t > T^ax, i-©- 



A{s) 



Ao{Tmax - for|s| < T, 
forlsl > 



max- 



Thus the signal at time t provides no information about the signal aAt — T for 
T > Tmax- With the triangular autocorrelation, we can compute the effective 
autocorrelation Tin (that can be seen as the time constant) of such an input: 

_ i^\s\A{s)ds 
!^A{s)ds 

which yields rj„ = in our simulations we explored values of Tin in the 

range of 3.3 < Tin < 80ms. 
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2.6 Spectral analysis 



For the interpretation of some of our results, it is useful to consider the power 
spectrum of the network activity. The spectrum is obtained by a fast Fourier 
transform on the temporal evolution of the spiking activity during 2^^ time 
steps. The spectrum is then averaged over 1000 frequency steps in order to 
smooth the curves. Such spectral curves give us the power at different frequen- 
cies. A single peak at a given frequency would indicate for example oscillations 
in the network at that precise frequency. As an artifact of the averaging pro- 
cedure sharp resonance peaks would be slightly broadened. We have checked 
that for the simulations shown in this paper, averaging does not cause artificial 
broadening. 



3 Results 



We stimulate a network of sparsely connected excitatory and inhibitory integrate- 
and-fire neurons by a continuous time-dependent input current (t) . We will 
evaluate for how long information about an input signal is kept in the net- 
work. To this end, we try to reconstruct the input signal at time t — T by 
looking at the instantaneous membrane potentials of all neurons at time t. If 
the reconstruction is possible, we say that the network is capable of "buffer- 
ing" information for a time T. The "buffering" performance as a function of 
T is the central quantity of interest. Information buffering is possible if the 
error of signal reconstruction is small. 

In the first subsection we investigate the role of noise on the performance 
measured in terms of the signal reconstruction error. We show that a mod- 
erate amount of noise systematically improves the performance. The role of 
asynchronous firing is emphasised. 

In the second subsection we focus on the temporal characteristics of the input 
signal. We derive an empirical relation between the temporal memory of the 
network and the effective autocorrelation of the input. 

Finally we show that the rich dynamics of the network in its asynchronous 
state provides a good representation of its past stimulation. Therefore simple 
linear readouts are powerful enough to extract information and perform nearly 
as well as advanced kernel-based methods [58,10]. 
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3.1 On the importance of noise 



3.1.1 Stochastic arrival of excitatory background spikes 

Networks of integrate-and-fire neurons are known to have complex dynam- 
ics [5,7]. In particular, our network of 200 excitatory and inhibitory neurons 
switches from highly synchronised neuronal activity to a very irregular firing 
regime, depending on the external input; figure 3 A and B, see also [5]. In the 
asynchronous phase, individual cells have a coefficient of variation Cy close 
to or larger than 1, reflecting this stochasticity (for the analytical computation 
of Cy, see [5]). The asynchronous firing phase can be reached with an exter- 
nal noise term corresponding to stochastic spike arrival at excitatory synapses 
(noise model A, see methods). The phase diagram of such a network is fully 
described in [5]. 

In order to characterise the network activity with this first model of noise 
(equation (4) see methods), we perform a preliminary series of simulations for 
different stochastic spike arrival rates v^xc at excitatory synapses in absence 
of a deterministic external stimulus. The power spectrum of the activity il- 
lustrates the regime of firing for the different drives, in particular the regimes 
of fast oscillations (above Uexc — ^OHz, c.g for Ug^c = 52.5Hz, see figure 2A) 
and irregular asynchronous firing (i^exc = 1 — 5Hz). In the asynchronous fir- 
ing phase, the power spectrum has no significant resonance peak (figure 2A, 
Uexc = 1.6Hz), the individual spike trains are irregular (figure 3A centre) and 
the overall activity has no oscillations (figure 3B centre) in absence of input 
signal. 

In a second series of simulation, we apply a time-dependent stimuhis current 
jext^ Neuronal activity refiects the temporal structure of the input (see figure 
3C). In order to assess whether we can estimate the input at time t — T from 
the set of membrane potentials at time t, we measure the signal reconstruction 
error (see methods). 

In figure 2B, short-term memory buffer performance is measured in terms of 
the reconstruction error on the test set for a delay T = 10ms by varying 
the rate Ug^c Ne — 100 excitatory synapses. Information buffer properties 
are good (i.e minimal reconstruction errors) up to Ugxc — '2Hz. The zone of 
optimality extends approximately up to the transition zone between quiescent 

^ This is a measure of the width of the distribution of spike intervals, defined as 
the ratio of the variance and the mean squared. A Poisson distribution has a value 
of Cv = 1. A value of Cy > 1, impUes that a given distribution is broader than a 
Poisson distribution with the same mean. If Cy < 1, then the spike train is more 
regular than that generated by a Poisson neuron of the same rate (see for example 
[26]). 
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and asynchronous irrcgTilar firing phase. In the quiescent regime (up to v^xc — 
IHz), there is no activity in absence of the input signal as seen in the 
left part of figure 3B ^ . However the neurons are close to threshold and the 
input signal, that is switched on at time 1000ms, make them fire when the 
input becomes sufficiently large, figure 3C left. For a weaker drive, neurons 
are farther from threshold and the input signal has to be stronger in order to 
ehcit a response. 

As the frequency Vexc is increased beyond IHz, the network reaches a phase of 
asynchronous firing. The population activity is now modulated by the input 
signal (figure 3C centre). 

Increasing the excitatory background input does not only increase the noise, 
but also the mean drive. A resonance peak is already present in the high 
frequency domain for a moderate drive (figure 2A, Uexc — 5Hz). The increased 
effective drive generates network activity at a high frequency (see the large 
peak at about 300Hz in figure 2A, i^exc = 52.5Hz). In such a regime, the 
individual cells fire at a large rate, figure 3 A right, and the overall activity 
has fast oscillations in absence of input signal, figure 3B right. The excitation 
by the time-dependent external signal P^*[t) is buried in the noise of the 
stochastic excitatory background, and the overall activity is barely modulated 
by the signal, figure 3C right. The performance hence decreases dramatically 
with an increase in the excitatory firing rate Ugxc- 

We can thus deduce from the error plot in figure 2B, that the network operates 
at best close to the stochastic dynamics, rather than in a network dominated 
by oscillations. 

Signature of chaos in neural systems is not a new concept. It was already 
shown in the past that networks of model neurons can give rise to a chaotic 
firing regime [29,62,66]. Experimental evidence indicates that cells exhibit in 
vivo very irregular spike trains, with a coefficient of variation CV close to 1 
(see e.g [61]), that may be the signature of a rich and very irregular underlying 
dynamics. From both an experimental and a theoretical perspective, the brain 
is thought to exhibit a chaotic behaviour [18,17,37]. 

In parallel, the fact that information processing by a dynamical system may 
gain from being close to a chaotic regime was postulated in the framework of 
cellular automata [38]. 

But only recently studies have focused on the fact that the brain may actually 
be operating at the edge of chaos [57,2]. Our findings suggest that networks of 

^ A more realistic readout, that would have access to the spikes of the individual 
neurons solely, would therefore only work when the network is not silent. Thus the 
zone of optimality would be dramatically narrowed. 
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spiking neurons can efficiently buffer information when close to a stochastic 
dynamics. 

Spontaneous stochastic activity at low rates is also known to be a prerequisite 
of rapid signaUing [25,6]. 

A B 




100 200 300 400 
Frequency [Hz] 



500 



0.8 

0.6 



Mo A 



, = 20 pA 

. I^Jj^ = 7.5 pA 
I Optimality Zone 



Quiescent State 




0.0001 



0.01 1 
V [Hz] 

exc ^ 



100 



Fig. 2. A: Power spectra of the population firing activity. B: Error as a function of 
Poisson noise. Notice the non-monotonic dependence on the amount of noise. As 
explained in [5], the asynchronous firing phase can be found for an intermediate drive 
(rather flat power spectrum). In a lower noise regime the neurons stand farther from 
threshold, the network remains silent in absence of a signal. Hence a spike-based 
readout would only become effective for an external drive larger than Uexc 

~ 0.8Hz. 

The zone of optimality is then located at the upper limit of the quiescent regime. 
In the high excitation limit, the mean drive dominates the effect of the "pure noise" 
component and oscillations appear. 



3.1.2 A relation to stochastic resonance: model B 

We can also set the network in an state of a stationary network activity with 
irregular individual spike trains by driving it with a balanced excitatory and 
inhibitory spike input (see methods, noise model B). As detailed in section 
2.2.2, this drive is equivalent to a constant input (contributed by the purely 
excitatory population only) and additional white noise (the variance parts 
of both the purely excitatory population and the balanced population). The 
performance is evaluated for different white noise amplitude on the top of a 
constant depolarisation oi fj,E — 0.6ml/. Errors on the test set are computed 
for different delays T (see figure 4A). As previously seen for the noise model 
A, there is a non-monotonic dependence of the performance upon the noise 
level. 

The signal reconstruction error is relatively high when the noise is low since 

the network is in a quiescent state. Addition of noise sets up the network in 
a state where the neurons are weakly coupled; each neuron act as an almost 
independent unit and its coupling to the other neurons enriches the network 
dynamics: we have reached a state in which the neurons are optimally desyn- 
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Fig. 3. A : Representative spike trains in absence of the time dependent external 
signal 7^^*(t), for the quiescent state (left), for the asynchronous state (centre) and 
for the synchronised state (right). B: Network activity and a representative noise 
signal for the quiescent state (left), for the asynchronous state (centre) and for the 
synchronised state (right). Note the different scale for the noise in the synchronised 
regime. C: The external input /^^*(t) has been switched on at time 1000ms (not 
shown on the graph). We show the network activity and the sum of the external 
input signal /^^*(t) and a representative noise signal for the quiescent state (left), 
for the asynchronous state (centre) and for the synchronised state (right). 

chronised. By adding more noise, the signal-to-noise ratio decreases along with 
the ability to retrieve information. The resonance peak present in the low noise 
case is smoothed out as the noise level increases^ (see figure 4B). In contrast 
to section 3.1.1, this non-monotonic dependence on the drive is now a pure 
noise effect, since the mean drive is kept constant. 



An additional interesting effect can be seen in figure 4A. For one delay T to 
another, the signal reconstruction error is at its lowest for a different noise level 
(referred as "optimal"). Surprisingly this optimal noise amplitude increases 

^ In the case of a larger network, the resonance peak would be washed out. Since 
our simulations include a small number of neurons, finite size effects (such as the 
impact of fluctuations) become strong. 
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Fig. 4. A: Errors as a function of the amplitude of noise in model B. Again, we 
can see that a moderate amount of noise level increases the performance of the 
network. In this case, this is purely a noise effect since the mean drive is constant. 
Thus it an also be interpreted as a stochastic resonance effect. Notice there is a 
different optimal noise level for the different delays T. A possible explanation is 
proposed in section 3.1.2. B: Corresponding power spectra of the activity. Since the 
constant drive is fixed, the noise just smoothes out the resonance peak present at 
a low noise level. Even though the peak in the power spectrum at high noise is 
slightly weakened, noise still has a corrupting effect on the input. The neurons are 
well desynchronised but the signal-to-noise ratio gets very low. 



with the buffering time T. It means that if we want to set up the network 
so that information remains available for a buffering time of 40ms, the noise 
has to be stronger than if we want to extract faithfully information after a 
buffering time of 10ms. A possible explanation is that in the low noise limit, 
transients are reflected almost instantaneously in the activity proflle [25,36]. 
A low noise level thus allows the readout to have a fine temporal resolution. In 
the high noise hmit however, the network acts as a low pass filter [25,6]. Slow 
transients for high noise allow longer buffering times. Therefore the amplitude 
of noise has to be higher for long buffering times than for short ones. 



Our results are related to the well-known phenomenon of stochastic resonance. 
Such an effect is seen when the response of a system to a drive depends non- 
monotonically on noise, with an optimum at a moderate, non-zero, noise level. 
There are many pointers in the literature to physical evidences of stochastic 
resonance in physical systems [69,11,23,24,47,71,70], and in models of neu- 
rons [8,42,55,54]. In living systems stochastic resonance has been reported in 
crayfish mechanoreceptors [16], the cricket cereal sensory system [40], neural 
slices [27], hippocampus [72], and the cortex [48]. More importantly the brain 
appears to actually make use of stochastic resonance at a functional level 
[35,30]. 
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3.2 Impact of autocorrelated inputs 



When the input is correlated in time, the readout can extract information 
from both the information buffered in the recurrent activity and from the in- 
put itself (filtered by the neurons). Thus if the autocorrelation of the input 
is short compared to the intrinsic temporal trace of the network, the role of 
the recurrency is crucial whereas for very long autocorrelation, the output will 
predominantly read "online" the input, making use of the "self-memory" ^ of 
the stimulation signal. We expect then to have an increase in performance as 
a function of the autocorrelation of the input. We can rule out an alternative 
explanation that a long autocorrelation has more time to impact the dynamics 
of the neural network by comparing the cross-correlation of the reconstructed 
output and the input with the signal autocorrelation for different autocorre- 
lation profiles (for an illustrated example and additional arguments, see [46]). 
Thus the cross- correlation depends on the shape and on the duration Tj„ of 
the signal autocorrelation. 

We performed numerical simulations using different effective autocorrelation 
times Tin for the input signals. The results are shown in figure 5A. We can 
estimate the effective information buffer by looking at the buffer time T at 
which the error rises to a value "arbitrary" chosen to be = 0.5 (the similarity 
of the reconstructed output to the target is still visually evident). 

As we can sec in figure 5B, the simulation points can be fitted to first ap- 
proximation by a straight line. In the limit of long autocorrelation rj„ ^ t„j 
we expect from simple mean field considerations, the buffer to have a tempo- 
ral extension longer than |rin, and to approach this line asymptotically from 
above ^ . Indeed the simulation results for large Tj„ stay always above the lower 
bound represented by the dashed line in figure 5B. 

For short autocorrelation profiles (of the order of the membrane time constant 



' On average the signal at time t holds information about itself at time f — T for 

T < T 

^ max ■ 

^ In order to derive a lower bound we could use the averaged membrane potential 
for reconstruction instead all the membrane potentials of the 200 neurons in the 
network. Since the distribution of the membrane potentials is modulated by the 
external input J*^^*, a reconstruction based on the average is possible. A trivial 
reconstruction can then be done assuming that the external input has not changed 
during the last time interval T. Assuming that the network can trivially buffer a 
signal that has not changed in the last time interval T, and that the network cannot 
reconstruct at all an input that has switched in this last time interval T, the effective 
buffer of the network for an input that changes every T^ax is T < ^Tmax- Since the 
effective autocorrelation of our input is Tj^ = ^T^^x: the network has an effective 
buffer of T = |ri„ 
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Tm = 20ms or shorter), this approximation no longer holds, since the filtering 
effects due to the neuron's membrane time constant comes into play. Rapid 
changes of the input on the scale of Tin ^ Tm will be averaged out irremedia- 
bly by the integration with the membrane time constant. Hence the network 
cannot buffer efficiently (i.e. with an error smaller than E — 0.5) input signals 
that have an effective autocorrelation Tin smaller than about 5ms (see figure 
5B). 

The performance can be improved for different autocorrelation profiles and for 
different network connectivities as proposed in [68] . They can also be increased 
by adding other time scales in the network, for example at the level of the 
synapses [45]. 

Analysis of cross-correlation and autocorrelation profiles (more precisely the 
analysis made in [46] of the location of the peak in the cross-correlation profile 
that is shifted in proportion of the autocorrelation length) show that the 
readout effectively makes an optimal trade-off between retrieving information 
from the the autocorrelation of the signal and from the buffer provided by the 
network recurrency. 
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Fig. 5. A: Error plots as a function of time for different input statistics. An evident 
increase in performance is seen as the autocorrelation of the input gets longer. B: 
Delays corresponding to an error of 0.5 as a function of the effective autocorrela- 
tion of the input. A linear fit shows us the clear dependence of the persistence of 
information in the network on the "self-memory" of the input. 



3.3 Simple linear readouts are efficient 



As we have seen in the section 3.1, injection of noise in the network helps in 
desynchronising the neurons. The resulting dynamics of the network is thus 
extremely rich. It has been shown that in the case of formal neural networks 
[2] and analogously for cellular automaton [38], the computational abilities of 
complex systems is at best when the system is close to a chaotic behaviour, as 
opposed to a dynamic state with short limit cycle. Another way of interpreting 
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this complex behaviour is to notice that the network provides a broad set of 
different fihcrs for the input. In other words, the one-dimensional input signal 
P^*{t) undergoes a "dimensional blow up". This is a strong correspondence 
to the working principle of some of the most efficient classification methods, 
named Support Vector Machines (SVM) [10]. These methods, known under 
the generic name of kernel methods [58] , send the input to a high dimensional 
space. In this multidimensional space, a simple linear separation (a hypcr-plan) 
into two sub-spaces is able to perform the classification. An adapted version of 
those methods can handle regression problems. In figure 6 a comparison of the 
performances of both the simple hnear readout and the SVM with a Gaussian 
kernel is represented (see methods). Although the kernel method achieves 




^0 20 40 60 80 100 

Delay T [ms] 

Fig. 6. Comparison of performances between Support Vector Machines (Gaussian 
kernel) and the simple linear readout on a test set. Optimisation has been performed 
on a separate training set of 50000 time steps. A simple linear combination of 
the membrane potentials is extremely effective in extracting information and is 
comparable to that of a Support Vector Machine. 



slightly better results than the linear combination, the gain is of the order of a 
few milliseconds only. The explanation is that an additional dimensional blow 

up that is at the basis of kernel methods is not necessary since a sufficiently 
high dimensional representation is already given by the network itself. Hence 
a simple linear combination (corresponding to the hyper-plan in the kernel 
methods) is able to extract most of the information needed. 
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4 Discussion and conclusions 



Wc investigated the buffering capacity of networks of excitatory and inhibitory 
integrate-and-fire neurons. The population firing activity of such a network 
with sparse connectivity can switch from an oscillatory firing state to an 
asynchronous irregular firing regime, depending on the rate of stochastic back- 
ground spike arrival. 



We found that the buffering capacity of the network is optimised in the pres- 
ence of a moderate, non-zero, amount of noise. This non-trivial dependence 
upon noise can be related to the well-known phenomenon of stochastic reso- 
nance. The optimal noise level, or equivalently the optimal discharge rate of 
an external population of neurons, corresponds approximately to the region 
of transition between a quiescent state and an asynchronous irregular firing 
regime. In this region, a very complex dynamics emerges from the network's 
activity. It provides a rich representation of the inputs. Simple adaptable lin- 
ear readouts are thus able to extract the buffered information in an efficient 
way. Hence our results provide an interpretation for the role of non-oscillatory 
dynamics in a simplified model of cortical micro-circuits. 



The buffering capacity of a such network can be generalised to the processing 
of a larger number of inputs. Simultaneous buffering of eight inputs is shown 
in figure 7 in comparison to a single input. No significant degradation is seen 
for a moderate number of inputs. A loss in performance can be seen only when 
processing more than sixteen inputs (data not shown). This ability to handle 
multiple simultaneous inputs, along with the way information can be extracted 
from both the autocorrelation of the input signals and the buffer provided 
by the network recurrency, could provide an interesting way of combining 
different sources of information having different time scales. On somewhat 
more speculative grounds, we can think of the cortex as being in a state that is 
close to a chaotic behaviour, and that other neighbouring neural micro-circuits 
or even farther afferents may be tuning the amount of chaos (by changing their 
spiking rate for example) of a given cortical micro-circuit depending on the 
relevent task. One can also imagine to build artificial networks made out of 
such model neurons, in order to do both information processing and prediction 
of (even chaotic) time series. The present study would suggest to set up such 
a device in a state where its dynamics can easily switch to a more complex 
behaviour, depending on its drive. 
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Fig. 7. Error plots for simultaneous buffering of eight inputs. A comparison to the 

bTiffcring of a single input (thick line) emphasises the fact that no significant degra- 
dation in performance is seen for a reasonable number of inputs. Our simulations 
show that a degradation is seen only from about sixteen simultaneous inputs. 
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